Tests: compare values, not their display (P32) - #45

«Tests» now compares values, not their display (P32). Only src/runtime/doctests.py changes: no protocol change, nothing in the macro or the IDE.

Behaviour

When the expected text and the output are each one Python literal (numbers, strings, lists, tuples, dicts, sets, True/False/None, nested), an example passes if ast.literal_eval(expected) == ast.literal_eval(got). So 4 for 4.0, [1,2,30] for [1, 2, 30] and ["+", 1, 2] for ['+', 1, 2] pass.

Everything else is compared as Python's doctest does, unchanged:

• exceptions;
• several lines of output;
• objects such as <function f at 0x…>;
• text that isn't a literal;
• set().

The report is unchanged too: Expected / Got show the texts as written. The counts, the coverage, the print hint and the «lacks blank after >>>» hint stay as they are.

Decisions

• Hook: _GlpythonOutputChecker(doctest.OutputChecker), passed to our DocTestRunner as checker. check_output first asks doctest's own check; only if that fails does it compare values. So nothing that passed before can fail now.
  • doctest also uses the checker for an expected exception (it compares the exception lines). Those lines aren't literals, so they stay as before.
  • output_difference (the report text) is doctest's own, unchanged.
• «One literal»: the text without its trailing newline must be one non-empty line, and ast.literal_eval must accept it. Any error (SyntaxError, ValueError, deep nesting, too many digits) means: no value comparison.
  • A list the learner wraps over two lines in the docstring is therefore compared as text. See the follow-ups.
• Option flags: if an example sets a comparison flag of its own (doctest.COMPARISON_FLAGS: ELLIPSIS, NORMALIZE_WHITESPACE, DONT_ACCEPT_TRUE_FOR_1, IGNORE_EXCEPTION_DETAIL, …), only doctest's check applies. Whoever sets a flag asked for doctest's comparison; DONT_ACCEPT_TRUE_FOR_1 in particular contradicts True == 1.
• print: what was printed is compared the same way as a displayed value, so print(5) now passes for 5.0, as return 5 does. Today print(5) and return 5 both pass for 5; that stays, and the print hint («probably return») still appears for every example that passes through print. Text output that isn't a literal (print("Hallo") → Hallo) is compared as text, as before.
• Floats: no tolerance. 0.3 for 0.1 + 0.2 fails, because == is False.
• True / 1: passes in both directions, because True == 1 in Python. doctest itself already accepted True for an expected 1. Also equal now: 1e3 / 1000, -0.0 / 0, 1_000 / 1000.
• A string isn't its content: return '4' for an expected 4 still fails ('4' != 4).
• IDE: the IDE runs the same worker from this package, so «Tests» behaves the same there once it pins the new tag (see below).

Checks

• check-runtime-doctests.py (in test:runtime) has a new section. It covers every case from P32 and the edge cases:

  Case	Result
  4 for 8 / 2; 5.0 for 5	pass
  [1,2,30] for [1, 2, 30]; "+" for '+'	pass
  ["+", 1, ["*", 2, 3]] for the same value with '…'	pass
  dict, tuple, set and None with other spacing	pass
  True for 1, and 1 for True	pass
  0.3 for 0.1 + 0.2; 4 for '4'	fail
  A wrong value	fails; want/got and Expected:/Got: show the texts as written
  An expected ZeroDivisionError	passes
  A different exception than the one expected	fails
  Several lines of output	text: same text passes, other spacing fails
  A list wrapped over two lines in the docstring	fails (text)
  print(5) for 5.0	passes, with the print hint
  print('Hallo') for Hallo / hallo	pass / fail (text)
  <function f at 0x0> for a function object	fails (text)
  set() for set()	passes
  {} for set()	fails
  +NORMALIZE_WHITESPACE with [1,2]	fails (doctest's check only)
  +ELLIPSIS with [1, ...]	passes
  +ELLIPSIS with 4 for 4.0	fails

• check-console-browser.mjs (test:console, Chrome with Pyodide):

  • The «Tests → pass» case now expects 5 for 5.0, so value comparison is also checked in Pyodide.
  • The «fail → edit» case needed a really wrong value (6 instead of 5 for 5.0), since 5 now passes.

• Locally green: lint (the known warning), test:protocol, test:runtime, test:hints, build:embed + test:console.

• README («Tests (doctests)»): new paragraph, and the example «Erhalten: 4.0 · Erwartet: 4» (which now passes) changed to «Erhalten: 4.5 · Erwartet: 4». CLAUDE.md updated.

How it gets live

• LiaScript courses: live without anything in the courses. The merge to main deploys dist-embed/ to GitHub Pages (deploy-embed.yml). The macro loads https://glpython-edu.github.io/glpython-editor/index.html live in its iframe, and the macro itself doesn't change, so no version bump is needed.
  • GitHub Pages sends Cache-Control: max-age=600, so a browser that had the embed open may keep the old one for up to 10 minutes (or until a reload after that).
  • The runtime is inlined in the worker bundle, whose file name has a hash, so the new index.html always brings the new runtime.
• IDE: needs a tag and a bump. After the merge, a tag on the merge commit (v0.1.21), then the tag in glpython/package.json (currently #v0.1.20). I'll prepare that PR in the IDE once the tag exists.

Manual tests

Embed (LiaScript, Etappe 2, after the deploy and a reload):

1. A block with >>> notenschnitt(4, 5, 6) / 5: «Tests» passes, «1 von 1».
2. >>> f() / [1,2,30] with return [1, 2, 30]: passes. With ["+", 1, 2] and return ['+', 1, 2]: passes.
3. A wrong value (>>> notenschnitt(4, 5, 6) / 6): fails, Expected: 6, Got: 5.0; «Beispiel visualisieren» shows «Erhalten: 5.0 · Erwartet: 6».
4. >>> 0.1 + 0.2 / 0.3: fails.
5. A profile with coverage (@Code.rezept3): all pass with 4 for 4.0 → the coverage still shows.
6. print instead of return (expected 5, print(5)): passes, with the print hint.
7. iPad: 1 and 3.

IDE (after the bump): 1–3 in a file of its own.

Follow-ups (not done here)

• Etappe 2, line 228 (materials/02-das-design-rezept/README.md) says the opposite: «Doctest vergleicht die Anzeige, nicht den Wert: Für Python ist 4.0 == 4 zwar True, … genau das muss unter dem Aufruf stehen.» Once this is live, that bullet is wrong. I didn't find other places in the course.
• A list wrapped over several lines in the docstring stays a text comparison (P32: «several lines … compare as today»). If learners write long expected lists over two lines (Etappe 4's nested lists?), that could be extended to «the whole expected text is one literal».
• Expected text without quotes (Hallo for return 'Hallo') still fails. That's a real type difference, not a display question; just mentioning it, since learners will meet it.

🤖 Generated with Claude Code